Papers with training time
Context-Aware Neural Machine Translation Decoding (D19-65)
Copied to clipboard
| Challenge: | Existing approaches to enhance neural machine translation systems to take into account document-level information make the training process slower or require document- level annotated data. |
| Approach: | They propose a decoding architecture that fuses the semantic space language model and a neural translation model. |
| Outcome: | The proposed approach improves translation quality for English–Spanish using BLEU and METEOR. |
Combating Security and Privacy Issues in the Era of Large Language Models (2024.naacl-tutorials)
Copied to clipboard
| Challenge: | a tutorial aims to provide a summary of risks and vulnerabilities in large language models . a number of studies have focused on security, privacy and copyright aspects of LLMs . |
| Approach: | This tutorial seeks to provide a systematic summary of risks and vulnerabilities in large language models . authors will discuss security, privacy and copyright aspects of LLMs . |
| Outcome: | This tutorial aims to provide a systematic summary of risks and vulnerabilities in large language models . it will also outline emerging challenges in security, privacy and reliability of LLMs . |
Rumor Detection on Twitter Using Multiloss Hierarchical BiLSTM with an Attenuation Factor (2020.aacl-main)
Copied to clipboard
| Challenge: | Existing models to classify rumors have low precision and are time consuming. |
| Approach: | They propose a multiloss hierarchical biLSTM model with an attenuation factor that can extract deep information from limited quantities of text. |
| Outcome: | The proposed model can extract deep information from limited quantities of text. |
Towards a Task-Agnostic Model of Difficulty Estimation for Supervised Learning Tasks (2020.aacl-srw)
Copied to clipboard
| Challenge: | Recent advances on natural language processing (NLP) benchmarks have been driven by increasingly sophisticated language models, which are pretrained on enormous amounts of data before use. |
| Approach: | They propose to develop a task-agnostic model for problem difficulty and apply it to the Stanford Natural Language Inference dataset. |
| Outcome: | The proposed model predicts how many annotators will answer a question correctly and then projectes the difficulty estimates onto the full SNLI train set to create the curriculum. |
Rethinking Full Finetuning from Pretraining Checkpoints in Active Learning for African Languages (2025.acl-srw)
Copied to clipboard
| Challenge: | Existing approaches to improve model performance are finetuning on all acquired data after each round, which is computationally expensive in multilingual and low-resource settings. |
| Approach: | They evaluate continual finetuning (CF) against full finetuned (FA) across 28 African languages using MasakhaNEWS and SIB-200. |
| Outcome: | The proposed approach outperforms full finetuning (FA) in 28 African languages, achieving up to 35% reductions in GPU memory, FLOPs, and training time. |
Bag-of-Words Transfer: Non-Contextual Techniques for Multi-Task Learning (D19-61)
Copied to clipboard
| Challenge: | Existing approaches to multi-task learning take advantage of transfer among tasks . generative reconstruction of the observations is not included in the standard framework . |
| Approach: | They propose to use a syntactically-oblivious pooling encoder and pre-trained word embeddings to improve sentence-level representations. |
| Outcome: | The proposed techniques yield similar performance on a universe of task combinations while reducing training time and model size. |
SpeedE: Euclidean Geometric Knowledge Graph Embedding Strikes Back (2024.findings-naacl)
Copied to clipboard
| Challenge: | Geometric knowledge graph embedding models (gKGEs) have shown great potential for knowledge graph completion (KGC) however, contemporary gKges require high embeddable dimensionalities or complex embeddances for good KGC performance, drastically limiting their time and space efficiency. |
| Approach: | They propose a lightweight Euclidean gKGE that provides strong inference capabilities and significantly outperforms state-of-the-art gGKGEs. |
| Outcome: | The proposed model outperforms state-of-the-art gKGEs on YAGO3-10 and WN18RR while significantly increasing their efficiency. |
Teach the Rules, Provide the Facts: Targeted Relational-knowledge Enhancement for Textual Inference (2021.starsem-1)
Copied to clipboard
| Challenge: | InferBERT is a method to enhance transformer-based inference models with relevant relational knowledge. |
| Approach: | They propose to enhance transformer-based inference models with relevant relational knowledge by injecting relevant facts at test time into the model. |
| Outcome: | The proposed method outperforms existing models on the challenge datasets while outperforming existing models. |
Lightweight Text Classifier using Sinusoidal Positional Encoding (2020.aacl-main)
Copied to clipboard
| Challenge: | Large and complex models require many parameters and time to solve various problems in natural language processing. |
| Approach: | They propose to use the sinusoidal positional encoding (SPE) to construct a convolutional neural network using the SPE in text classification. |
| Outcome: | The proposed model reduces parameter size and training time while maintaining similar performance to the current model on multiple benchmark datasets. |
Learning Sparsity for Effective and Efficient Music Performance Question Answering (2025.acl-short)
Copied to clipboard
| Challenge: | Existing Music AVQA methods rely on dense and unoptimized representations, leading to inefficiencies in the isolation of key information, reduction of redundancy, and prioritization of critical samples. |
| Approach: | They propose a sparse learning framework specifically designed for Music AVQA to address these challenges. |
| Outcome: | The proposed framework reduces training time by 28.32% while maintaining accuracy while maintaining state-of-the-art performance on the Music AVQA datasets. |
Local-to-global learning for iterative training of production SLU models on new features (2022.naacl-industry)
Copied to clipboard
| Challenge: | In many real-world NLP systems, new data becomes available with time and there is a need to refresh the model. |
| Approach: | They propose to adapt a local-to-global learning schedule to production settings where full data is not available at initial training iterations. |
| Outcome: | The proposed model improves model error rates by 7.3% and saves up to 25% training time for individual iterations. |
Improving Mongolian-Chinese Neural Machine Translation with Morphological Noise (P19-2)
Copied to clipboard
| Challenge: | Existing models for Mongolian-Chinese translation are based on recurrent, convolutional neural networks or completely eliminate recurrence connections. |
| Approach: | They propose a adversarial training model to alleviate the UNK problem in Mongolian-Chinese machine translation by adding a screener to the model to emphasize the added Mongolian morphological noise. |
| Outcome: | The proposed model reduces training time and improves accuracy in Mongolian-Chinese translation tasks. |
Data Pruning for Efficient Model Pruning in Neural Machine Translation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models have demonstrated encouraging performance in various NLP tasks at the cost of over-parametrized networks and high memory requirements. |
| Approach: | They combine data pruning with movement pruning for Neural Machine Translation to enable efficient fine-pruning by leveraging cross-entropy scores of individual training instances. |
| Outcome: | The proposed pruning strategy outperforms other pruning methods on a translation task and shows that training cross-entropy scores can reduce the steps required for convergence and training time. |
Mitigating the Burden of Redundant Datasets via Batch-Wise Unique Samples and Frequency-Aware Losses (2023.acl-industry)
Copied to clipboard
Donato Crisostomi, Andrea Caciolai, Alessandro Pedrani, Kay Rottmann, Alessandro Manzotti, Enrico Palumbo, Davide Bernardi
| Challenge: | Existing solutions to train deep learning models on redundant datasets are difficult to implement in industrial settings. |
| Approach: | They propose a method to eliminate duplicates at the batch level without altering the data distribution observed by the model. |
| Outcome: | The proposed approach reduces training times on models on redundant datasets by up to 87% and 46% on average, with a drop in model performance of 0.2% relative at worst. |
Learning to Generate Word- and Phrase-Embeddings for Efficient Phrase-Based Neural Machine Translation (D19-56)
Copied to clipboard
| Challenge: | Neural machine translation (NMT) often fails in one-to-many translation, e.g., in the translation of multi-word expressions, compounds, and collocations. |
| Approach: | They propose a phrase-based NMT model that generates embeddings of words or phrases. |
| Outcome: | The proposed model performs on par with state-of-the-art phrase-based NMT. |
SmBoP: Semi-autoregressive Bottom-up Semantic Parsing (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing semantic parsers decode syntax using a top-down depth-first traversal. |
| Approach: | They propose a semi-autoregressive bottom-up parser that constructs at decoding step t the top-K sub-trees of height t. |
| Outcome: | The proposed method achieves 2.2x speed-up in decoding time and 5x speed up in training time on a zero-shot semantic parsing benchmark. |
HybridBERT - Making BERT Pretraining More Efficient Through Hybrid Mixture of Attention Mechanisms (2024.naacl-srw)
Copied to clipboard
| Challenge: | Pretrained transformer-based language models have produced state-of-the-art performance in most natural language understanding tasks. |
| Approach: | They propose two hybrid architectures that combine self-attention and additive attention mechanisms with sub-layer normalization to achieve double the pretraining accuracy of a vanilla-BERT baseline. |
| Outcome: | The proposed architectures outperform BERT-base on two downstream tasks while accelerating inference. |
Simple Large-scale Relation Extraction from Unstructured Text (L18-1)
Copied to clipboard
| Challenge: | Knowledge-based question answering relies on the availability of facts, most of which cannot be found in structured sources. |
| Approach: | They propose a method for creating distant (weak) supervision labels for training a large-scale RE system by decoupling the model architecture from the feature design of a state-of-the-art neural network system. |
| Outcome: | The proposed method performs on par with the state-of-the-art model with similar features at 75x reduction in training time. |
Overlapping Context with Variable-Length Stride Increases Diversity when Training Large Language Model for Code (2025.acl-industry)
Copied to clipboard
Geonmo Gu, Jaeho Kwak, Haksoo Moon, Hyun Seung Shim, Yu Jin Kim, Byoungjip Kim, Moontae Lee, Hyejeong Jeon
| Challenge: | Large language models for code (LLMs) are gaining more and more attention due to their wide applicability. |
| Approach: | They propose a method which extracts overlapping contexts from training data using variable-length stride. |
| Outcome: | The proposed method outperforms the conventional approach of controlling the number of epochs in terms of the pass@k rate. |
Practical Takes on Federated Learning with Pretrained Language Models (2023.findings-eacl)
Copied to clipboard
| Challenge: | federated learning with pretrained language models for language tasks entails data privacy constraints when learning from diverse data domains. |
| Approach: | They propose to use pretrained language models to learn from diverse data domains . they elaborate hypotheses over the components in federated NLP architectures based on three tasks . |
| Outcome: | The proposed model can generalize by adapting to the different domains. |
R-LoRA: Randomized Multi-Head LoRA for Efficient Multi-task Learning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) improves performance in multi-task learning by diversifying the head matrices through Multi-Head Dropout and Multi-head Random Initialization. |
| Approach: | They propose a low-rank adaptive approach to fine-tune large language models by approximating weight updates through low-ranked matrices. |
| Outcome: | The proposed approach improves performance in multi-task learning while reducing memory usage and training time. |
Split-NER: Named Entity Recognition via Two Question-Answering-based Classifications (2023.acl-short)
Copied to clipboard
| Challenge: | Named entity recognition (NER) is a foundational task for a variety of applications like question answering and machine translation. |
| Approach: | They propose to split entity recognition problem into two sub-tasks and optimize them separately for each sub-task. |
| Outcome: | The proposed system outperforms baselines on OntoNotes5.0, WNUT17 and a cybersecurity dataset and gives on-par performance on BioNLP13CG. |
Hyperbolic Geometry is Not Necessary: Lightweight Euclidean-Based Models for Low-Dimensional Knowledge Graph Embeddings (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent knowledge graph embedding models based on hyperbolic geometry are complicated than Euclidean operations. |
| Approach: | They propose to use hyperbolic geometry to generate high-fidelity and parsimonious representations of hierarchical patterns in knowledge graphs. |
| Outcome: | The proposed models achieve state-of-the-art performance on two widely-used datasets and cost less than RotH. |
Improving Training Efficiency and Reducing Maintenance Costs via Language Specific Model Merging (2026.eacl-industry)
Copied to clipboard
| Challenge: | Recent research on merging multilingual multitask models has shown promise in terms of improved quality, but its computational and maintenance efficiency remains unstudied. |
| Approach: | They propose a method for fine-tuning a multilingual large language model . they compare a "retrain-all" approach and a 'train-once, merge-as-needed' approach . |
| Outcome: | The proposed approach reduces training time by up to 50% while maintaining parity in terms of quality. |
Does Self-Attention Need Separate Weights in Transformers? (2025.naacl-industry)
Copied to clipboard
| Challenge: | Experimental results show a 66.53% reduction in parameter size within the attention block and competitive accuracy improvements of 3.55% and 0.89% over symmetric and pairwise attention-based models, respectively. |
| Approach: | They propose a simplified approach where a single weight matrix is used for Keys, Queries, and Values instead of separate matrices for each. |
| Outcome: | The proposed approach outperforms the BERT baseline on GLUE tasks even outperforming the standard BERT model in handling noisy and out-of-domain data. |
Domain Generalisation of NMT: Fusing Adapters with Leave-One-Domain-Out Training (2022.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in neural machine translation (NMT) research have found NMT models sensitive to distribution shift and adversarial examples. |
| Approach: | They propose a leave-one-domain-out training strategy that learns to combine domain-specific parameters to avoid information leaking. |
| Outcome: | The proposed method outperforms baselines on three language pairs on average. |
Scheduled Sampling for Transformers (P19-2)
Copied to clipboard
| Challenge: | Existing studies show that scheduled sampling can be applied to recurrent neural networks to avoid exposure bias. |
| Approach: | They propose to use teacher forced embeddings and model predictions to avoid exposure bias in sequence-to-sequence generation. |
| Outcome: | The proposed technique achieves performance close to a teacher-forcing baseline on two language pairs and is promising for future research. |
Named Entity Recognition for Chinese biomedical patents (2020.coling-main)
Copied to clipboard
| Challenge: | Existing attempts to address NER for Chinese biomedical texts have been limited due to the amount of Chinese biomedicine discoveries being patented. |
| Approach: | They train and evaluate Chinese biomedical patents NER models based on BERT . their model is optimized for Chinese bio-patent data and scored an F1 . |
| Outcome: | The proposed model achieves an F1 score of 0.540.15 for Chinese biomedical patent data. |
Correctable-DST: Mitigating Historical Context Mismatch between Training and Inference for Improved Dialogue State Tracking (2022.emnlp-main)
Copied to clipboard
Hongyan Xie, Haoxiang Su, Shuangyong Song, Hao Huang, Bo Zou, Kun Deng, Jianghua Lin, Zhihui Zhang, Xiaodong He
| Challenge: | Existing dialogue state tracking approaches predict the dialogue state of a target turn sequentially based on the ground-truth previous dialogue state. |
| Approach: | They propose a method that predicts dialogue state sequentially based on previous dialogue state . they propose generating a previously “predicted” dialogue state using ground-truth previous dialogue states . |
| Outcome: | The proposed method achieves 67.51%, 68.24%, 70.30%, 71.38%, and 81.27% joint goal accuracy on MultiWOZ 2.0-2.4 datasets. |
Does the Order of Training Samples Matter? Improving Neural Data-to-Text Generation with Curriculum Learning (2021.eacl-main)
Copied to clipboard
| Challenge: | Recent advances in data-to-text generation have been focused on curriculum learning, which is a process of presenting training data in a specific order, starting from easy examples and moving on to more difficult ones, as the learner becomes more competent. |
| Approach: | They propose to use a curriculum learning process to change the order of training samples in a model based on the model's competence to improve model performance and convergence speed. |
| Outcome: | The proposed model shows faster convergence speed and reduced training time by 38.7% and performance by 4.84 BLEU. |
A Framework for Flexible Extraction of Clinical Event Contextual Properties from Electronic Health Records (2025.acl-industry)
Copied to clipboard
| Challenge: | EHRs contain vast amounts of valuable clinical data, stored as unstructured text. |
| Approach: | They propose a method that uses existing NER+L methods to classify medical entities at scale using a named entity recognition and linking task. |
| Outcome: | The proposed model outperforms Bi-LSTM in minority class tasks with up to 28% of the time and 32% faster training time. |
Sub-Word Alignment is Still Useful: A Vest-Pocket Method for Enhancing Low-Resource Machine Translation (2022.acl-short)
Copied to clipboard
| Challenge: | Low-resource machine translation (MT) is challenging due to the scarcity of parallel data and lack of bilingual dictionaries. |
| Approach: | They propose to leverage embedding duplication between aligned sub-words to extend the Parent-Child transfer learning method to improve low-resource machine translation. |
| Outcome: | The proposed method achieves BLEU scores of 22.5, 28.0 and 18.1 respectively. |
Re-evaluating the Need for Visual Signals in Unsupervised Grammar Induction (2024.findings-naacl)
Copied to clipboard
Boyi Li, Rodolfo Corona, Karttikeya Mangalam, Catherine Chen, Daniel Flaherty, Serge Belongie, Kilian Weinberger, Jitendra Malik, Trevor Darrell, Dan Klein
| Challenge: | Recent studies show multimodal inputs can improve grammar induction, but weak textual baselines are needed for training. |
| Approach: | They use a fixed grammar family to compare multimodal grammar induction methods . they find multimodal inputs can improve grammar induction by grounding textual inputs to the visual world . |
| Outcome: | The proposed model outperforms weaker baselines on four benchmark datasets. |
Extending Neural Generative Conversational Model using External Knowledge Sources (D18-1)
Copied to clipboard
| Challenge: | Existing generative dialogue models lack coherence and are content poor . however, current models lack the capacity to handle large unstructured knowledge sources. |
| Approach: | They propose an architecture to incorporate unstructured knowledge sources to enhance the next utterance prediction in chit-chat type of generative dialogue models. |
| Outcome: | The proposed architecture improves the next utterance prediction in chit-chat type of generative dialogue models by incorporating external knowledge from Wikipedia summaries and the NELL knowledge base. |
AdaPT: A Set of Guidelines for Hyperbolic Multimodal Multilingual NLP (2024.findings-naacl)
Copied to clipboard
| Challenge: | Euclidean space is used for training neural models and performing arithmetic operations, but many data types have complex geometries and cannot be captured in the Euclidesan space. |
| Approach: | They propose a set of guidelines for initialization, parametrization, and training of neural networks that can be generalized over existing neural network training methodologies. |
| Outcome: | The proposed framework outperforms Euclidean methods on three tasks over 12 languages and modalities on a variety of domains. |
Domain Knowledge Transferring for Pre-trained Language Model via Calibrated Activation Boundary Distillation (2022.acl-long)
Copied to clipboard
| Challenge: | Pretrained language models are used to boost their performance on downstream tasks . pretraining with in-domain texts requires considerable in- domain data and training resources . |
| Approach: | They propose a domain knowledge transferring framework for pre-trained language models without additional in-domain pretraining. |
| Outcome: | The proposed framework extracts domain knowledge from an existing in-domain pretrained language model and transfers it to other PLMs by applying knowledge distillation. |
Competence-based Curriculum Learning for Neural Machine Translation (N19-1)
Copied to clipboard
| Challenge: | Existing NMT systems require specialized heuristics and large batch sizes. |
| Approach: | They propose a curriculum learning framework for NMT that reduces training time and costs . framework consists of a principled way of deciding which training samples are shown to the model . |
| Outcome: | The proposed framework can reduce training time and improve performance of recurrent neural network models and Transformers. |
A Progressive Model to Enable Continual Learning for Semantic Slot Filling (D19-1)
Copied to clipboard
| Challenge: | Existing approaches to slot filling training on large scale data are inefficient and require multiple trainings. |
| Approach: | They propose a slot filling model that transfers previously learned knowledge to a small size expanded component and enables it to be fast trained to learn from new data. |
| Outcome: | The proposed model outperforms existing models on two benchmark datasets by 4.24% and 3.03% on the same dataset. |
Sequicity: Simplifying Task-oriented Dialogue Systems with Single Sequence-to-Sequence Architectures (P18-1)
Copied to clipboard
| Challenge: | Existing solutions to task-oriented dialogue systems follow pipeline designs which introduces complexity and fragility. |
| Approach: | They propose a novel sequence-to-sequence (seq2sequ) model which tracks dialogue believes and a two stage copynet instantiation which emonstrates good scalability. |
| Outcome: | The proposed framework outperforms state-of-the-art pipeline-based methods on large datasets and retains satisfactory entity match rate on out-of vocabulary (OOV) cases where pipeline-designed competitors totally fail. |
Joint Multi-Label Attention Networks for Social Text Annotation (N19-1)
Copied to clipboard
| Challenge: | Present research shows that title metadata could affect social annotation. |
| Approach: | They propose a title-guided attention network for document annotation with user-generated tags that separates the title from the content of a document and applies a semantic-based loss regulariser over each sentence in the content. |
| Outcome: | The proposed approach outperforms the Bi-GRU and Hierarchical Attention Network (HAN) on two open datasets with 10%-30% reduction in training time. |
Zero-Shot End-to-End Spoken Language Understanding via Cross-Modal Selective Self-Training (2024.eacl-long)
Copied to clipboard
| Challenge: | End-to-end (E2E) spoken language understanding models are constrained by the cost of collecting speech-semantics pairs. |
| Approach: | They propose a model that learns E2E SLU without speech-semantics pairs . they propose cross-modal selective self-training (CMSST) to address imbalance and noise issues . |
| Outcome: | The proposed model learns E2E SLU without speech-semantics pairs . the proposed model requires the domains of speech-text and text-sensitization to match . |
Training Long-Context LLMs Efficiently via Chunk-wise Optimization (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in long-context large language models have demonstrated superior retrieval quality compared to retrievalaugmented generation (RAG) approaches. |
| Approach: | They propose a memory-efficient training paradigm that partitions lengthy inputs into manageable chunks. |
| Outcome: | The proposed model expands maximum sequence length from 1K to 16K tokens on a single RTX 3090 GPU, while SpaCO achieves accelerated training speed. |
NeuroPrune: A Neuro-inspired Topological Sparse Training Algorithm for Large Language Models (2024.findings-acl)
Copied to clipboard
Amit Dhurandhar, Tejaswini Pedapati, Ronny Luss, Soham Dan, Aurelie Lozano, Payel Das, Georgios Kollias
| Challenge: | Transformer-based Language Models have become ubiquitous in natural language processing due to impressive performance on various tasks. |
| Approach: | They explore how sparsity affects network topology by exploiting mechanisms seen in biological networks . they show that model-agnostic sparsities are performant across diverse NLP tasks . |
| Outcome: | The proposed model-agnostic sparsity approaches are performant and efficient across NLP tasks. |
Improved Language Modeling by Decoding the Past (P19-1)
Copied to clipboard
| Challenge: | Existing methods to improve language modeling performance are based on regularized LSTMs with a large number of parameters and training time. |
| Approach: | They propose a method that decodes the last token in context using the predicted distribution of the next token. |
| Outcome: | The proposed method improves perplexity on the Penn Treebank dataset by 1.8 points and 2.3 points on the WikiText-2 datasets. |
OpenHands: Making Sign Language Recognition Accessible with Pose-based Pretrained Models across Languages (2022.acl-long)
Copied to clipboard
| Challenge: | a new study examines the performance of pretraining for sign language recognition in low-resource settings. |
| Approach: | They propose using pose extracted through pretrained models as the standard modality of data to reduce training time and enable efficient inference. |
| Outcome: | The proposed model reduces training time and allows efficient inference in sign languages. |
Accelerating Neural Transformer via an Average Attention Network (P18-1)
Copied to clipboard
| Challenge: | Using parallelizable attention networks, the neural Transformer is slow to train due to auto-regressive architecture and self-attention in the decoder. |
| Approach: | They propose an average attention network to replace the original self-attention model in the decoder of the neural Transformer. |
| Outcome: | The proposed network can decode sentences over four times faster than the original version with almost no loss in training time and translation performance. |
EarlyBERT: Efficient BERT Training via Early-bird Lottery Tickets (2021.acl-long)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models require enormous computational resources and long training time. |
| Approach: | They propose an algorithm to reduce inference time and train large NLP models by slimming the self-attention and fully-connected sub-layers inside a transformer. |
| Outcome: | The proposed algorithm achieves comparable performance to standard BERT with 35 45% less training time. |
Transformer-based Models for Long-Form Document Matching: Challenges and Empirical Analysis (2023.findings-eacl)
Copied to clipboard
| Challenge: | Recent advances in the area of long document matching have primarily focused on using transformer-based models for long document encoding and matching. |
| Approach: | They propose to use simple neural models and simple embeddings to improve document matching by taking significantly less training time, energy, and memory. |
| Outcome: | The proposed models outperform the more complex BERT-based models while taking significantly less training time, energy, and memory. |
Exploring Phoneme-Level Speech Representations for End-to-End Speech Translation (P19-1)
Copied to clipboard
| Challenge: | Previous work on end-to-end translation from speech uses frame-level features as speech representations, which creates longer, sparser sequences than text. |
| Approach: | They propose a method to generate compressed phoneme-like speech representations that generate shorter, higher-level source sequences for translation. |
| Outcome: | The proposed method improves translation performance by 5 BLEU on high and low resource languages and reduces training time by 60%. |
On the Effects of Transformer Size on In- and Out-of-Domain Calibration (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Large pre-trained transformer language models are notoriously expensive to train . prior work has developed smaller, more compact models to reduce training costs . |
| Approach: | They propose to develop smaller, more compact transformer language models which can be calibrated in-domain . they show that smaller models can achieve competitive calibration compared to larger models . |
| Outcome: | The proposed models achieve competitive calibration and better calibration than larger models on a wide range of tasks. |
PVPO: Pre-Estimated Value-Based Policy Optimization for Agentic Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | extending grouping-based methods to agentic reasoning presents unique challenges . frequent environment interactions and tool invocations render intra-group advantage estimation unstable . |
| Approach: | They propose a grouping-based method that uses a single round of rollouts to stabilize advantage estimation. |
| Outcome: | a new RL framework outperforms grouping-based methods in retrieval tasks and advanced mathematical reasoning benchmarks. |
DynaMo: Accelerating Language Model Inference with Dynamic Multi-Token Sampling (2024.naacl-long)
Copied to clipboard
| Challenge: | Rapid explosion in model sizes has resulted in high inference times . open-source LLMs are democratizing research in natural language processing . |
| Approach: | They propose a suite of multi-token prediction language models that reduce net inference times by leveraging traditional autoregressive weights. |
| Outcome: | The proposed model achieves same-quality generated text as baseline (Pythia-6.9B) with only 5.87% and 2.67% parameter and training time overheads. |
Highly Efficient Knowledge Graph Embedding Learning with Orthogonal Procrustes Analysis (2021.naacl-main)
Copied to clipboard
| Challenge: | Knowledge Graph Embeddings (KGEs) have been explored in recent years due to their promise for a wide range of applications. |
| Approach: | They propose a KGE framework which can reduce the training time and carbon footprint by orders of magnitudes compared with state-of-the-art approaches. |
| Outcome: | The proposed framework reduces the training time and carbon footprint by orders of magnitudes compared with state-of-the-art approaches while producing competitive performance. |
AdaRewriter: Unleashing the Power of Prompting-based Conversational Query Reformulation via Test-Time Adaptation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Prompting-based conversational query reformulation has emerged as a powerful approach for conversational search, refining ambiguous user queries into standalone search queries. |
| Approach: | They propose a framework for query reformulation using an outcome-supervised reward model via test-time adaptation. |
| Outcome: | Experiments on five conversational search datasets show that AdaRewriter significantly outperforms the existing methods across most settings. |
Adaptive Gating in Mixture-of-Experts based Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models employ a fixed gating network where each token is computed by the same number of experts. |
| Approach: | They propose a flexible training strategy that allows tokens to be processed by a variable number of experts based on expert probability distribution. |
| Outcome: | The proposed model reduces training time and inference quality while maintaining sparsity while maintaining inference accuracy. |
Show Your Work: Improved Reporting of Experimental Results (D19-1)
Copied to clipboard
| Challenge: | Current practice is to train multiple instantiations of each, choose the best model of each type, and compare their performance on held-out test data. |
| Approach: | They propose to measure expected validation accuracy as a function of computation budget . authors find comparisons where authors would have reached different conclusions if they had used more computation . |
| Outcome: | The proposed method shows that test-set performance scores alone are insufficient for drawing accurate conclusions about which model performs best. |
A Batch Normalized Inference Network Keeps the KL Vanishing Away (2020.acl-main)
Copied to clipboard
| Challenge: | Variational Autoencoder (VAE) is widely used to approximate a model’s posterior on latent variables. |
| Approach: | They propose to let the Kullback–Leibler divergence individual follow a distribution across the whole dataset and analyze that it is sufficient to prevent posterior collapse by keeping the expectation of the KL’s distribution positive. |
| Outcome: | The proposed approach can avoid posterior collapse effectively and efficiently without introducing any new model component or modifying the objective. |
Hi-ZFO: Hierarchical Zeroth- and First-Order LLM Fine-Tuning via Importance-Guided Tensor Selection (2026.findings-acl)
Copied to clipboard
| Challenge: | generative tasks require a high degree of exploratory capacity, but zeroth-order methods suffer from slow convergence . generative task-specific methods tend to converge toward local minima, causing noise and inefficient estimation . |
| Approach: | They propose a framework that synergizes FO precision with exploratory capability of ZO estimation. |
| Outcome: | The proposed framework synergizes precision of FO gradients with exploratory capability of ZO estimation. |
Scalable Font Reconstruction with Dual Latent Manifolds (2021.emnlp-main)
Copied to clipboard
| Challenge: | a recent study has shown that fonts with a large number of missing glyphs are difficult to model due to the relative sparsity of most fonts. |
| Approach: | They propose a deep generative model that performs typography analysis and font reconstruction by learning disentangled manifolds of both font style and character shape. |
| Outcome: | The proposed model scales up the number of character types we can model compared to previous methods . it can generalize to characters that were not observed during training time, and it compares favorably to other models . |
Sparse Low-rank Adaptation of Pre-trained Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning pre-trained large language models in a parameter-efficient manner are gaining traction within the research community. |
| Approach: | They propose a method of low-rank adaptation that enables dynamic adjustments to the intrinsic rank during the adaptation process. |
| Outcome: | The proposed approach outperforms the current method with a fixed and unalterable intrinsic rank and a low-rank adaptation process. |
Target Conditioning for One-to-Many Generation (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Neural Machine Translation models lack diversity in their generated translations, even when paired with search algorithm, like beam search. |
| Approach: | They propose to model one-to-many mapping by conditioning a decoder on a latent variable that represents the domain of target sentences. |
| Outcome: | The proposed method can scale to any number of domains without affecting performance or training time. |
Improving Transformer Models by Reordering their Sublayers (2020.acl-main)
Copied to clipboard
| Challenge: | a sandwich transformer pattern is a new approach to multilayer transformers that can be used for different tasks. |
| Approach: | They propose a transformer ordering pattern that reorders sublayers in a sandwich transformer pattern . they generate random transformer models and train them with the language modeling objective . |
| Outcome: | The proposed pattern improves perplexity on multiple word-level and character-level language modeling benchmarks at no cost in parameters, memory, or training time. |
ACE-M3: Automatic Capability Evaluator for Multimodal Medical Models (2025.coling-main)
Copied to clipboard
| Challenge: | Existing metrics for multimodal large language models only focus on token overlap and may not align with human judgment. |
| Approach: | They propose an open-source model that assesses the question answering abilities of multimodal large language models. |
| Outcome: | Experiments show that the ACE-M3 model performs better than existing models and is more reliable than existing metrics. |
DARER: Dual-task Temporal Relational Recurrent Reasoning Network for Joint Dialog Sentiment Classification and Act Recognition (2022.findings-acl)
Copied to clipboard
| Challenge: | Dialog sentiment classification (DSC) and dialog act recognition (DAR) aims to predict the sentiment label and act label for each utterance in a dialog. |
| Approach: | They propose a framework which integrates prediction-level interactions other than semantics-level ones into dialog understanding and dual-task reasoning by integrating temporal relations into the model. |
| Outcome: | The proposed model outperforms existing models by large margins while costing less training time and requiring less computation resource. |
On Transferability of Prompt Tuning for Natural Language Processing (2022.naacl-main)
Copied to clipboard
Yusheng Su, Xiaozhi Wang, Yujia Qin, Chi-Min Chan, Yankai Lin, Huadong Wang, Kaiyue Wen, Zhiyuan Liu, Peng Li, Juanzi Li, Lei Hou, Maosong Sun, Jie Zhou
| Challenge: | Pre-trained language models (PLMs) can achieve comparable performance to full-parameter fine-tuning by tuning a few soft prompts, but require much more training time than fine-timing. |
| Approach: | They empirically investigate the transferability of soft prompts across different downstream tasks and PLMs to determine what decides prompt transferability. |
| Outcome: | The proposed method can achieve comparable performance to full-parameter fine-tuning by tuning a few soft prompts, but requires much more training time than fine-timing. |
Mid-Think: Training-Free Intermediate-Budget Reasoning via Token-Level Triggers (2026.findings-acl)
Copied to clipboard
Van Yang, Shouren Wang, Debargha Ganguly, Xinpeng Li, Chaoda Song, Vikash Singh, Vipin Chaudhary, Xiaotian Han
| Challenge: | Explicit /think> tags are used to expose intermediate reasoning and enable hybrid thinking behaviors. |
| Approach: | They propose a training-free prompting format that combines these triggers to achieve intermediate-budget reasoning, outperforming fixed-token and prompt-based baselines in terms of the accuracy–length trade-off. |
| Outcome: | The proposed method outperforms fixed-token and prompt-based prompts in accuracy–length trade-offs while improving Qwen3-8B on AIME from 69.8% to 72.4% and on GPQA from 58.5% to 61.1%. |
Class-based LSTM Russian Language Model with Linguistic Information (2020.lrec-1)
Copied to clipboard
| Challenge: | LSTM models can be used in speech recognition systems at N-best or lattice rescoring stage. |
| Approach: | They propose to use word frequency and linguistic information to generate class-based LSTM Russian language models with various numbers of classes. |
| Outcome: | The proposed models outperform word-based models and word2vec models in terms of perplexity, training time, and word error rate. |
Sparse Growing Transformer: Training-Time Sparse Depth Allocation via Progressive Attention Looping (2026.findings-acl)
Copied to clipboard
Yao Chen, Yilong Chen, Yinqi Yang, Junyuan Shang, Zhenyu Zhang, Zefeng Zhang, Shuaiyi Nie, Shuohuan Wang, Yu Sun, Hua Wu, Haifeng Wang, Tingwen Liu
| Challenge: | Existing approaches to increasing effective depth of LLMs rely on parameter reuse, extending computation through recursive execution. |
| Approach: | They propose a training-time sparse depth allocation framework that progressively increases depth for a small subset of parameters as training evolves. |
| Outcome: | The proposed model outperforms existing approaches to increasing the effective depth of language models while reducing training FLOPs overhead from approximately 16–20% to only 1–3% relative to a standard Transformer backbone. |
Multi-Hop Transformer for Document-Level Machine Translation (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing approaches to document-level neural machine translation (NMT) simply introduce the representations of context sentences without explicitly characterizing the inter-sentence reasoning process. |
| Approach: | They propose a novel multi-hop Transformer which explicitly models the human-like draft-editing and reasoning process by attending to multiple antecedent sentences iteratively. |
| Outcome: | Experiments on four widely used document translation tasks show that the proposed model significantly improves document-level translation performance and tackles discourse phenomena such as coreference error and the problem of polysemy. |
LogicST: A Logical Self-Training Framework for Document-Level Relation Extraction with Incomplete Annotations (2024.emnlp-main)
Copied to clipboard
| Challenge: | Document-level relation extraction (DocRE) is difficult due to the vast number of entity pairs. |
| Approach: | They propose a neural-logic self-training framework that iteratively resolves conflicts and constructs the minimal diagnostic set for updating models. |
| Outcome: | The proposed framework outperforms existing methods on the document-level relation extraction (docRE) benchmark. |
Data Quality Enhancement on the Basis of Diversity with Large Language Models for Text Classification: Uncovered, Difficult, and Noisy (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods for text classification based on large language models are difficult to apply directly to solve. |
| Approach: | They propose a data quality enhancement method to improve LLMs' performance in classification tasks by using a greedy algorithm to select data and then performing fine-tuning. |
| Outcome: | The proposed method improves the performance of large language models in text classification tasks and significantly improves training efficiency, saving nearly half of the training time. |
Comparison of Conventional Hybrid and CTC/Attention Decoders for Continuous Visual Speech Recognition (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent advances have been achieved in Visual Speech Recognition (VSR) despite the lack of data, there is no clear comparison between different types of decoders for certain languages and tasks. |
| Approach: | They focused on how the conventional DNN-HMM decoder behaves depending on the amount of data used for their estimation. |
| Outcome: | The proposed model improves the CTC/Attention model in data-scarcity scenarios while requiring less training time and fewer parameters. |
Disentangling Preference Representation and Text Generation for Efficient Individual Preference Alignment (2025.coling-main)
Copied to clipboard
| Challenge: | Human values are inherently diverse, making it insufficient to align LLMs solely with general preferences. |
| Approach: | They propose a flexible paradigm for individual preference alignment that disentangles preference representation from text generation in LLMs. |
| Outcome: | The proposed method produces aligned quality and better than PEFT-based methods while reducing training time for each new individual preference by 80% to 90%. |
Generative Deduplication For Socia Media Data Selection (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Social media data exhibits severe redundancy due to its noisy nature, leading to increased training time and model bias in its processing. |
| Approach: | They propose a new framework for deduplication of social media data by removing semantically duplicate data from the model and add time-dimensional Gaussian noise to reduce training complexity. |
| Outcome: | The proposed framework can reduce training samples while improving performance over baselines. |
ARCH: Efficient Adversarial Regularized Training with Caching (2021.findings-emnlp)
Copied to clipboard
Simiao Zuo, Chen Liang, Haoming Jiang, Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, Tuo Zhao
| Challenge: | Existing approaches to regularize models require generating a perturbation for each sample in each epoch. |
| Approach: | They propose an adversarial regularization method where perturbations are generated and cached once every several epochs. |
| Outcome: | The proposed method significantly eases the computational burden (saves up to 70% of computational time) it produces a notably better (in most of the tasks) or comparable model generalization. |
Multi-Reference Training with Pseudo-References for Neural Translation and Text Generation (D18-1)
Copied to clipboard
| Challenge: | Neural text generation has been quite successful recently, but during training time, only one reference is considered for each example, even though there are often multiple references available. |
| Approach: | They propose an algorithm to generate exponentially many pseudo-references by compressing existing references into lattices and traversing them to generate new pseudo-References. |
| Outcome: | The proposed model significantly improves on baselines in machine translation and image captioning, and is comparable to existing models. |
Structural Contrastive Representation Learning for Zero-shot Multi-label Text Classification (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches for zero-shot multi-label text classification struggle with accuracy and poor training efficiency. |
| Approach: | They propose a structural contrastive representation learning approach that uses randomized text segmentation to generate high-quality contrastive pairs. |
| Outcome: | The proposed approach improves accuracy and speed up training time on publicly available datasets. |
Enhanced Language Representation with Label Knowledge for Span Extraction (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to extract text spans from plain text do not fully exploit label knowledge. |
| Approach: | They propose a model to integrate label knowledge into text representations by encoding texts and annotations independently and then integrating label knowledge with an elaborate-designed semantics fusion module. |
| Outcome: | The proposed model achieves state-of-the-art performance on four benchmarks and reduces training time and inference time by 76% and 77% on average compared with the existing paradigm. |
Learning Better Masking for Better Language Model Pre-training (2023.acl-long)
Copied to clipboard
| Challenge: | Existing PrLMs adopt a Random-Token Masking strategy with a fixed masking ratio and different contents are masked by an equal probability throughout the training. |
| Approach: | They propose two scheduled masking approaches that adaptively tune masking ratio and masked content in different training stages, which improves pre-training efficiency and effectiveness. |
| Outcome: | The proposed methods improve the pre-training efficiency and effectiveness on the downstream tasks. |
The Lower The Simpler: Simplifying Hierarchical Recurrent Models (N19-1)
Copied to clipboard
| Challenge: | Using a simplified version of GRU, we replace the GRUs at the middle layers of hierarchical recurrent models with Fixed-size Ordinally-Forgetting Encoding (FOFE). |
| Approach: | They propose to make the lower layers simpler than the upper ones to simplify two typical hierarchical recurrent models, namely Hierarchical Recurrent Encoder-Decoder (HRED) and R-NET, whose basic building block is GRU. |
| Outcome: | The proposed models contain less trainable parameters, consume less training time, and achieve slightly better performance than baseline models. |
Direct Simultaneous Speech-to-Text Translation Assisted by Synchronized Streaming ASR (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to simultaneous speech-to-text translation suffer from error propagation and extra latency. |
| Approach: | They propose a new paradigm for simultaneous speech-to-text translation using two separate decoders . they use multitask learning to jointly learn these two tasks with a shared encoder . |
| Outcome: | The proposed method achieves substantially better translation quality at similar levels of latency. |
Phrase-level Self-Attention Networks for Universal Sentence Encoding (D18-1)
Copied to clipboard
| Challenge: | Phrase-level self-attention networks (PSAN) can capture context dependencies at the phrase level instead of the sentence level. |
| Approach: | They propose to perform self-attention across words inside a phrase to capture context dependencies at the phrase level and use the gated memory updating mechanism to refine each word’s representation hierarchically with longer-term context dependency captured in a larger phrase. |
| Outcome: | The proposed model can achieve state-of-the-art performance across a plethora of NLP tasks including binary and multi-class classification, natural language inference and sentence similarity. |
End-to-End Slot Alignment and Recognition for Cross-Lingual NLU (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to expand an NLU system to new languages use machine translation with slot label projection from source to translated utterances. |
| Approach: | They propose a novel end-to-end model that learns to align and predict target slot labels jointly for cross-lingual transfer. |
| Outcome: | The proposed model outperforms a simple label projection method on most languages and achieves competitive performance to the more complex, state-of-the-art projection method with only half the training time. |
Scale Down to Speed Up: Dynamic Data Selection for Reinforcement Learning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Current approaches to Reinforcement Learning (RL) rely on massive static datasets, leading to computational inefficiency and redundant gradient updates. |
| Approach: | They propose a data-centric RL framework that dynamically selects the most informative training samples to optimize RL for mathematical reasoning. |
| Outcome: | The proposed framework achieves comparable performance to full-data training methods while requiring only 1.5K samples instead of 220K, reducing training time from 13 days to just 4 hours on 8A800 GPUs. |
Shortformer: Better Language Modeling using Shorter Inputs (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods require computationally expensive relative position embeddings. |
| Approach: | They propose two methods that decrease input length to improve perplexity and perplexability. |
| Outcome: | The proposed methods speed up training by a factor of 1.65 and reduce memory usage. |
Do It Once: An Embarrassingly Simple Joint Matching Approach to Response Selection (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing matching models for response selection perform the independent matching (IM) approach. Existing models for matching only perform one match regardless of the number of options. |
| Approach: | They propose a joint matching approach which performs matching only once regardless of the number of options. |
| Outcome: | The proposed approach outperforms existing models and reduces training time by over half. |
JParaCrawl: A Large Scale Web-Based English-Japanese Parallel Corpus (2020.lrec-1)
Copied to clipboard
| Challenge: | Recent machine translation algorithms rely on parallel corpora, but only some resource-rich language pairs can benefit from them. |
| Approach: | They construct a parallel corpus for English-Japanese, which has 8.7 million sentence pairs . they use a web crawler to automatically align parallel sentences in the corpus . |
| Outcome: | The proposed corpus includes a broader range of domains and can be trained with a pre-trained model. |
Leveraging Denoised Abstract Meaning Representation for Grammatical Error Correction (2023.findings-acl)
Copied to clipboard
| Challenge: | Popular GEC models use large-scale synthetic corpora or use a large number of human-designed rules. |
| Approach: | They propose a model that incorporates denoised AMR as additional knowledge to get AMRs more reliable. |
| Outcome: | The proposed model reduces training time by 32% while inference time is comparable. |
Effects of sub-word segmentation on performance of transformer language models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Language models are a fundamental task in natural language processing, but few studies focus on the effect of sub-word segmentation on the performance of models. |
| Approach: | They compare GPT and BERT models trained with statistical segmentation algorithm BPE to unsupervised morphological segmentation algorithms Morfessor and StateMorph. |
| Outcome: | The proposed model trains for several languages and compares them with two unsupervised morphological segmentation algorithms. |
MMR-GRPO: Accelerating GRPO-Style Training through Diversity-Aware Reward Reweighting (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have demonstrated remarkable capabilities in mathematical reasoning tasks. |
| Approach: | They propose to use Maximal Marginal Relevance to reweigh rewards of multiple rollouts by balancing rollout quality with diversity to reduce rollout redundancy. |
| Outcome: | The proposed approach reduces training time and costs by 47.9% . evaluations across three model sizes, three GRPO variants, and five mathematical reasoning benchmarks show that it achieves comparable peak performance while requiring on average 70.2% less wall-clock time. |
Not All Adapters Matter: Selective Adapter Freezing for Memory-Efficient Fine-Tuning of Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning pre-trained models impose substantial resource usage. |
| Approach: | They propose a parameter-efficient fine-tuning method that freezes adapters early to reduce resource usage while maintaining performance. |
| Outcome: | The proposed method reduces memory usage, computation amount, and training time by 42.85%, 34.59%, and 11.82% while maintaining performance. |
Learning Universal Sentence Representations with Mean-Max Attention Autoencoder (D18-1)
Copied to clipboard
| Challenge: | Existing methods to learn universal sentence representations focus on supervised learning. |
| Approach: | They propose a mean-max attention autoencoder that uses a multi-head mechanism to reconstruct the input sequence. |
| Outcome: | The proposed model outperforms state-of-the-art unsupervised single methods on a wide range of 10 transfer tasks. |
Federated Learning of Large Language Models with Parameter-Efficient Prompt Tuning and Adaptive Optimization (2023.emnlp-main)
Copied to clipboard
| Challenge: | Prompt tuning of Large Language Models (LLMs) can incur performance degradation or low training efficiency. |
| Approach: | They propose a prompt tuning approach with Adaptive Optimization to enable efficient FL of LLMs. |
| Outcome: | The proposed approach improves performance and efficiency simultaneously and addresses client drift problems on both the device and server sides. |
Learning Programmatic Idioms for Scalable Semantic Parsing (D19-1)
Copied to clipboard
| Challenge: | In state-of-the-art semantic parsers map natural language instructions to source code . idioms improve the accuracy of semantic parses, allowing for faster decoding . |
| Approach: | They propose an iterative method to extract code idioms from large source code corpora . they use most-frequent subtrees of their syntax trees to train semantic parsers to apply them . |
| Outcome: | The proposed method improves the state-of-the-art semantic parsers' accuracy and training time by more than 50%. |
Cache-Efficient Posterior Sampling for Reinforcement Learning with LLM-Derived Priors Across Discrete and Continuous Domains (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are used as action proposers in reinforcement learning (RL) but they are expensive and require prohibitive computational costs. |
| Approach: | They propose a cache-efficient framework for Bayesian RL that leverages large language models as action proposers and optimizes meta-learning based on policy performance to enable efficient inference across text-based games and robotic control tasks. |
| Outcome: | The proposed framework achieves 3.8–4.7 reduction in LLM queries and 4.0–12.0 lower median latencies (85–93ms on consumer hardware) while retaining 96–98% of the uncached policy’s performance. |
Document-Level Multi-Event Extraction with Event Proxy Nodes and Hausdorff Distance Minimization (2023.acl-long)
Copied to clipboard
| Challenge: | Document-level multi-event extraction aims to extract the structural information from a given document automatically. |
| Approach: | They propose an alternative approach for document-level multi-event extraction with event proxy nodes and Hausdorff distance minimization. |
| Outcome: | The proposed method outperforms state-of-the-art methods on two datasets with only a fraction of training time. |
Improved Unsupervised Chinese Word Segmentation Using Pre-trained Knowledge and Pseudo-labeling Transfer (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to unsupervised Chinese word segmentation require multiple inferences to perform word segmenting. |
| Approach: | They propose a method that integrates the segmentation signal from an unsupervised language model to a pre-trained BERT classifier under a pseudo-labeling framework. |
| Outcome: | The proposed method achieves state-of-the-art performance on the eight UCWS tasks while significantly reducing training time compared to previous approaches. |
Revisiting Token Dropping Strategy in Efficient BERT Pretraining (2023.acl-long)
Copied to clipboard
| Challenge: | Token dropping is a recently-proposed strategy to speed up the pretraining of masked language models, such as BERT. |
| Approach: | They propose a semantic-consistent learning method to improve token dropping by skipping the computation of a subset of input tokens at several middle layers. |
| Outcome: | The proposed method achieves consistent and significant performance gains across all tasks and model sizes. |
GGP: Glossary Guided Post-processing for Word Embedding Learning (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing word embedding models require much training time and domain knowledge to improve. |
| Approach: | They propose a GGP-based word embedding model that incorporates the glossary and learns sense representations. |
| Outcome: | The proposed model outperforms existing models on topical/functional similarity datasets by 4.1% and 7%. |
Distilling the Essence: Efficient Reasoning Distillation via Sequence Truncation (2026.findings-acl)
Copied to clipboard
Wei-Rui Chen, Vignesh Kothapalli, Ata Fatahibaarzi, Hejian Sang, Shao Tang, Qingquan Song, Zhipeng Wang, Muhammad Abdul-Mageed
| Challenge: | a large reasoning model (LRM) training on large amounts of reasoning data is computationally expensive. |
| Approach: | They propose a method to quantify computation-quality tradeoffs as a function of sequence length. |
| Outcome: | The proposed method reduces training time, memory and FLOPs by 50% on long training sequences while retaining the full-sequence performance. |
Continual Learning in Task-Oriented Dialogue Systems (2021.emnlp-main)
Copied to clipboard
Andrea Madotto, Zhaojiang Lin, Zhenpeng Zhou, Seungwhan Moon, Paul Crook, Bing Liu, Zhou Yu, Eunjoon Cho, Pascale Fung, Zhiguang Wang
| Challenge: | Existing continuous learning systems are not designed to add new domains and functionalities through time without incurring the high cost of retraining the whole system. |
| Approach: | They propose a first-ever continual learning benchmark for task-oriented dialogue systems . they propose 'architecture' method based on residual adapters to implement continual training . |
| Outcome: | The proposed architectural method performs better than multitask learning while being 20X faster in learning new domains. |
Debiasing knowledge graph embeddings (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to train knowledge graph embeddings to be neutral to sensitive attributes such as gender have been shown to increase training time by a factor of eight or more. |
| Approach: | They propose a method where all embeddings are trained to be neutral to sensitive attributes such as gender by default using an adversarial loss. |
| Outcome: | The proposed method reduces training time by eightfold and improves accuracy. |
Autoregressive Reasoning over Chains of Facts with Transformers (2020.coling-main)
Copied to clipboard
| Challenge: | Existing work on this task either evaluates facts in isolation or artificially limits the possible chains of facts, thus limiting multi-hop inference. |
| Approach: | They propose an iterative inference algorithm that decomposes the selection of facts from a corpus autoregressively and conditioning the next iteration on previously selected facts. |
| Outcome: | The proposed method outperforms the previous state-of-the-art in terms of precision, training time and inference efficiency on the WorldTree dataset. |
HFT: Half Fine-Tuning for Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) with one or more fine-tuning phases can unlock various capabilities, but can be catastrophic forgetting during sequential training. |
| Approach: | They propose a method to regularly reset partial parameters to mitigate forgetting issues by using half fine-tuning instead of full fine-uning. |
| Outcome: | The proposed approach reduces the risk of catastrophic forgetting during training and the parametric knowledge lost during training may be overwhelmed by incoming training data. |
Unveiling the Role of Pretraining in Direct Speech Translation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to train direct speech-to-text translation systems are pretraining the encoder on automatic speech recognition, thus losing efficiency in the training process. |
| Approach: | They propose to change the decoder cross-attention to integrate source information from earlier steps in training. |
| Outcome: | The proposed model can achieve comparable performance to the pretrained model while reducing training time. |
SLAM: Towards Efficient Multilingual Reasoning via Selective Language Alignment (2025.coling-main)
Copied to clipboard
Yuchun Fan, Yongyu Mu, YiLin Wang, Lei Huang, Junhao Ruan, Bei Li, Tong Xiao, Shujian Huang, Xiaocheng Feng, Jingbo Zhu
| Challenge: | Large language models (LLMs) have demonstrated significant improvements in reasoning abilities, but these improvements are primarily focused on English, leading to inferior performance in non-English scenarios. |
| Approach: | They propose a multilingual reasoning alignment approach that fine-tunes the layers responsible for multilingual comprehension in one stage. |
| Outcome: | The proposed method fine-tunes 6 of the 9 layers responsible for multilingual comprehension, while reducing training time by 4.1-11.9 compared to the two-stage method. |
VIP5: Towards Multimodal Foundation Models for Recommendation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in foundation models have impeded the ability for these fields to benefit from each other’s advancements. |
| Approach: | They propose to use a multimodal foundation model to unify various modalities and recommendation tasks under the P5 recommendation paradigm to implement personalized prompts. |
| Outcome: | The proposed model will unify visual, textual, and personalization modalities under the P5 recommendation paradigm and will improve recommendation performance and efficiency. |
Context-based Virtual Adversarial Training for Text Classification with Noisy Labels (2022.lrec-1)
Copied to clipboard
| Challenge: | Recent studies show that deep neural networks can memorize noisy labels with limited training time. |
| Approach: | They propose a virtual adversarial training method to prevent a classifier from overfitting to noisy labels. |
| Outcome: | The proposed method performs the adversarial training in the context rather than the inputs. |
Rethinking and Improving Multi-task Learning for End-to-end Speech Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | auxiliary tasks are highly consistent with end-to-end speech translation (ST) but their effectiveness has not been thoroughly studied. |
| Approach: | They propose an improved multi-task learning approach for the ST task that bridges the modal gap by mitigating the difference in length and representation. |
| Outcome: | The proposed approach achieves state-of-the-art on the MuST-C dataset with 20.8% of training time required by the current SOTA method. |
SoftRegex: Generating Regex from Natural Language Descriptions using Softened Regex Equivalence (D19-1)
Copied to clipboard
| Challenge: | Existing models that generate semantically correct regular expressions from NLs are not yet fully understood. |
| Approach: | They propose a model that rewards reinforcement learning based on the semantic equivalence between two regular expressions. |
| Outcome: | The proposed model reduces training time and produces state-of-the-art results on three benchmark datasets. |
CaPE: Contrastive Parameter Ensembling for Reducing Hallucination in Abstractive Summarization (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing work suggests that the degree of hallucination depends on factual errors in training data. |
| Approach: | They propose a method to use training data to reduce hallucination by ensembling parameter variations in training data. |
| Outcome: | The proposed method improves on XSUM and CNN/DM datasets on human evaluations and factual metrics. |
DaMoC: Efficiently Selecting the Optimal Large Language Model for Fine-tuning Domain Tasks Based on Data and Model Compression (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models excel in general tasks but struggle with domain-specific ones, requiring fine-tuning with specific data. |
| Approach: | They propose a Data and Model Compression Framework that categorizes data filtering methodologies into three distinct paradigms: (1) distribution-aware methods, (2) quality-a aware methods, and (3) hybrid approaches considering both dimensions. |
| Outcome: | The proposed framework can select the optimal LLM while saving approximately 20-fold in training time. |
How Do Hyenas Deal with Human Speech? Speech Recognition and Translation with ConfHyena (2024.lrec-main)
Copied to clipboard
| Challenge: | Currently, attention-based models face computational hurdles in processing long sequences due to its quadratic complexity. |
| Approach: | They propose a conformer whose encoder self-attentions are replaced with Hyena for speech processing . they propose 'confhyena' model that reduces training time by 27% at minimal cost . |
| Outcome: | The proposed model reduces training time by 27% at the cost of minimal quality degradation. |
Hyperbolic Representations for Prompt Learning (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing techniques to train only continuous prompts while freezing the language model have been developed. |
| Approach: | They propose to use hyperbolic space to model hierarchical relationships between prompts and inputs . they use a Poincaré disk to capture the hierarchic relationship between prompt and input . |
| Outcome: | The proposed approach reduces training time and storage for downstream tasks by reducing training costs. |
Indic-TEDST: Datasets and Baselines for Low-Resource Speech to Text Translation (2024.lrec-main)
Copied to clipboard
| Challenge: | Speech-to-text Translation (ST) tasks are performed by human translators with proficiency in both the source and target languages. |
| Approach: | a new study compares the performance of SOTA ST models on low-resource languages . the authors propose to use a dataset to compare the models on high-resourced languages based on the results of their research . |
| Outcome: | a new study shows that only a few models have performed well on low-resource languages . the results indicate the need for specialized models for low- and high-resourced languages based on the dataset . |
Token-level Preference Self-Alignment Optimization for Multi-style Outline Controllable Generation (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing attempts to outline generation are limited by response pair requirements and substantial computation costs. |
| Approach: | They propose a token-level preference self-alignment optimization for outline controllable generation that extends the Bradley-Terry model from pair-wise to list-wise comparison. |
| Outcome: | The proposed method outperforms existing methods by 19.28% in performance while requiring only 56.25% training time. |
Adapting Where It Matters: Depth-Aware Adaptation for Efficient Multilingual Speech Recognition in Low-Resource Languages (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent speech foundation models excel at multilingual automatic speech recognition (ASR) for high-resource languages, but their performance drops substantially on low-resourced languages due to the limited data availability. |
| Approach: | They propose a Depth-Aware Model Adaptation framework that allocates adaptation capacity according to each layer’s role. |
| Outcome: | The proposed framework matches or surpasses state-of-the-art accuracy with 80% fewer trainable parameters and achieves 29% error reduction under extreme data scarcity. |
Context Minimization for Resource-Constrained Text Classification: Optimizing Performance-Efficiency Trade-offs through Linguistic Features (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained language models have transformed text classification, but their computational demands often render them impractical for resource-constrained settings. |
| Approach: | They propose a linguistically-grounded framework for context minimization that leverages theme-rheme structure to preserve critical classification signals while reducing input complexity. |
| Outcome: | The proposed framework preserves critical classification signals while reducing input complexity. |
Efficient Data Learning for Open Information Extraction with Pre-trained Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Experimental results indicate that, compared to previous SOTA methods, OK-IE requires only 1/100 of the training data (900 instances) and 1/120 of the time (3 minutes) to achieve comparable results. |
| Approach: | They propose a framework that transforms OpenIE into the pre-training task form of the T5 model, thereby reducing the need for extensive training data. |
| Outcome: | The proposed framework transforms OpenIE into the pre-training task form of the T5 model, reducing the need for extensive training data and significantly reducing training time. |
Sparsity-Accelerated Training for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated proficiency across various NLP tasks but often require additional training, such as continual pre-training and supervised fine-tuning. |
| Approach: | They propose to leverage sparsity in pre-trained LLMs to accelerate training by disregarding computations for unimportant neurons. |
| Outcome: | The proposed framework achieves comparable or superior performance to standard training while significantly accelerating the process. |
VE-KD: Vocabulary-Expansion Knowledge-Distillation for Training Smaller Domain-Specific Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | VE-KD is a method that balances knowledge distillation and vocabulary expansion with the aim of training efficient domain-specific language models. |
| Approach: | They propose a method that balances knowledge distillation and vocabulary expansion with the aim of training efficient domain-specific language models. |
| Outcome: | VE-KD outperforms DistilBERT and Adapt-and-Distill in biomedical domain tasks . compared with other methods, it outperformed Distilbert and adapted-and distill . |
DEFT: Distribution-guided Efficient Fine-Tuning for Human Alignment (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Experimental results show that the methods enhanced by DEFT outperform the original methods in both alignment capability and generalization ability, with significantly reduced training time. |
| Approach: | They propose a distribution-based alignment framework that integrates data filtering and distributional guidance to improve alignment efficiency and generalization ability. |
| Outcome: | The proposed framework outperforms existing methods in alignment capability and generalization ability with significantly reduced training time. |
Beyond the Context Window: Scaling Agentic RL via End-to-end Optimized Context Compression (2026.acl-long)
Copied to clipboard
| Challenge: | Existing reinforcement learning pipelines suffer from degraded instruction following, excessive rollout costs, and strict context limits. |
| Approach: | They propose a reinforcement learning (RL) fine-tuning of large language model (LLM) agents for long-horizon multi-turn tool use where context length quickly becomes a bottleneck. |
| Outcome: | The proposed framework improves the success rate while maintaining the same or even lower working context length compared to baselines. |
TransferCVLM: Transferring Cross-Modal Knowledge for Vision-Language Modeling (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent large vision-language multimodal models pre-trained with huge amount of image-text pairs show remarkable performances in downstream tasks. |
| Approach: | They propose a method of efficient knowledge transfer that integrates pre-trained uni-modal models into a combined vision-language model without pre-training . they propose to fine-tune the model and transfer multimodal knowledge from a teacher vision-linguistic model to the CVLM for each task application. |
| Outcome: | The proposed method outperforms existing vision-language models in downstream tasks. |
EcoLoRA: Communication-Efficient Federated Fine-Tuning of Large Language Models (2025.emnlp-main)
Copied to clipboard
Han Liu, Ruoyao Wen, Srijith Nair, Jia Liu, Wenjing Lou, Chongjie Zhang, William Yeoh, Yevgeniy Vorobeychik, Ning Zhang
| Challenge: | Recurrent exchange of model updates in FL can result in prohibitively high communication costs, hindering the distributed learning process. |
| Approach: | They propose a federated fine-tuning framework that uses a round-robin segment sharing scheme to reduce network bandwidth and adaptive sparsification methods tailored to LoRA’s training dynamics. |
| Outcome: | The proposed framework reduces communication overhead without compromising performance on question-answering and value-alignment tasks. |
R-BPE: Improving BPE-Tokenizers with Token Reuse (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large pretrained language models prioritize high-resource languages in their vocabularies, leaving others with poor coverage. |
| Approach: | They propose a framework that reuses existing tokenizers and creates ID-based maps to resolve the new tokens of the chosen language. |
| Outcome: | The proposed framework reduces subword fertility by 24.4% on Arabic models and preserves performance on EnglishMMLU. |
TinyJudge: Unverifiable Constraint Alignment via Lightweight Specialist Ensembles (2026.acl-long)
Copied to clipboard
Yirong Zeng, Yufei Liu, Xiao Ding, Yutai Hou, Yuxian Wang, Wu Ning, Haonan Song, Dandan Tu, Qixun Zhang, Yuxiang He, Bibo Cai, Ting Liu
| Challenge: | Instruction Following (IF) is a core capability of LLMs, requiring strict adherence to diverse constraints. |
| Approach: | They propose a framework that uses tiny language models to evaluate instruction following . they propose to use a set of specialized tiny language model to provide rewards for soft constraints. |
| Outcome: | The proposed framework outperforms baseline models by 12% and speeds up training time by 3. |
Understand the Implication: Learning to Think for Pragmatic Understanding (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods rely on annotated labels but overlook the reasoning process humans naturally use to interpret implicit meaning. |
| Approach: | They propose a dataset that includes explicit reasoning for both correct and incorrect interpretations and propose supervised fine-tuning to improve their performance. |
| Outcome: | The proposed dataset improves LLMs' pragmatic understanding by 11.12% across model families and 16.10% over label trained models. |
HomoGraphAdapter: A Homogeneous Graph Neural Network as an Effective Adapter for Vision-Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing adaptation methods overlook structural knowledge between text and image modalities or create overly complex graphs containing redundant information for alignment. |
| Approach: | They propose a method to adapt visual models to downstream tasks using text and image modalities. |
| Outcome: | The proposed method improves classification accuracy by 1.51% for 1-shot and 0.74% for 16-shot on 11 datasets. |
Self-Knowledge Distillation for Knowledge Graph Embedding (2024.lrec-main)
Copied to clipboard
| Challenge: | Knowledge graph embedding (KGE) is an important task for many downstream applications. |
| Approach: | They propose to use self-knowledge distillation to learn a low-dimensional model from a pre-trained high-dimensional one. |
| Outcome: | The proposed model can improve model performance while maintaining lightweight structure. |
Studying the Role of Input-Neighbor Overlap in Retrieval-Augmented Language Models Training Efficiency (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have shown that retrieval-augmented language models can perform better than larger models, but the optimal degree of overlap remains unclear. |
| Approach: | They investigate how varying levels of query–context overlap affect model performance . they specifically generate synthetic context through paraphrasing queries . |
| Outcome: | The proposed model can improve test-time perplexity and training time by 40% without compromising performance. |
Activating Distributed Visual Region within LLMs for Efficient and Effective Vision-Language Training and Inference (2025.acl-long)
Copied to clipboard
| Challenge: | Existing Large Vision-Language Models (LVLMs) learn visual capacity through visual instruction tuning. |
| Approach: | They propose a method for LVLMs to be trained by selective layers tuning . they propose removing non-critical layers outside the visual region . |
| Outcome: | The proposed approach preserves nearly 99% of visual performance and improves textual task results while reducing training time. |
Less is More: Explainable and Efficient ICD Code Prediction with Clinical Entities (2025.acl-long)
Copied to clipboard
| Challenge: | Clinical coding is labor-intensive and prone to delays, leading to global backlogs. |
| Approach: | They propose an approach that combines Named Entity Recognition (NER) and Assertion Classification (AC) to filter for clinically important content before supervised code prediction. |
| Outcome: | The proposed approach reduces training time by over half on a standard evaluation dataset compared to current methods . it uses Named Entity Recognition (NER) and Assertion Classification (AC) to filter for clinically important content before supervised code prediction. |
Accelerating Training of Autoregressive Video Generation Models via Local Optimization with Representation Continuity (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing autoregressive models have shown superior performance and efficiency in image generation, but are constrained by high computational costs and prolonged training times in video generation. |
| Approach: | They propose a Local Optimization method which optimizes tokens within localized windows while leveraging contextual information to reduce error propagation. |
| Outcome: | The proposed method achieves superior performance to the baseline while halving the training cost without sacrificing quality. |
DPLoRA: A Dual-Pruning Framework based on ILP Optimization and Progressive Pruning for Parameter-Efficient LoRA Fine-Tuning (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) require computational resources for fine-tuning. |
| Approach: | They propose a framework that optimizes rank allocation via two stages . they propose an initial pruning stage and a progressive pruning stage . |
| Outcome: | The proposed framework outperforms existing PEFT baselines on GLUE and instruction-following tasks while reducing training time and trainable parameters by over 80%. |
Learning Faster with Better Tokens: Parameter-Efficient Vocabulary Adaptation for Specialized Text Summarization (2026.acl-long)
Copied to clipboard
| Challenge: | a new approach to adapt generalist models to expert domains is needed to overcome this problem. |
| Approach: | They propose a parameter-efficient domain adaptation approach that combines vocabulary adaptation with pretraining for LLM-based text summarization. |
| Outcome: | The proposed approach reduces training time by 35-55% over continual pretraining and reduces parameter counts up to 37% w.r.t expansion-only methods. |